Description of work

Stream bluesky documents to kafka. This enables remote, out-of-process callbacks to trigger from bluesky scans or data-points.

See https://reduce.isis.cclrc.ac.uk/redpanda-console/topics/NDW2922_bluesky?p=-1&s=50&o=-1#messages for an example of msgpack-encoded documents which have been streamed from my dev machine.

This is using the upstream bluesky-kafka integration. It looks like that implementation is somewhat used by one or two facilities, but much less than core bluesky. If the upstream implementation causes a problem later, it would be pretty trivial to write our own (direct) wrapper that forwards messages to confluent-kafka via msgpack directly.

Ticket

None

Labels

• 'bluesky-Semver-Minor' - New feature

Acceptance criteria

• Pull request title is understandable for a user (e.g. scientist) reading the release notes. The PR title should be a short description of the change from a user perspective.
• Pull request has appropriate labels for automatic release-notes generation
• Bluesky documents streamed to kafka unconditionally
• Kafka being unavailable does not cause errors or a scan to fail (messages should simply buffer in the producer). e.g. test this by pulling network cable on your computer mid-scan, scan should continue unaffected.

Documentation

Kafka docs added in 'developer' section.

Not expecting this to be user-facing, so no user-facing docs added.